System and method for multi-image-based vessel proximity situation recognition support

ABSTRACT

A system and method for multi-image-based vessel proximity situation recognition support is proposed. The system may include an unmanned surface vehicle (USV) configured to detect and track surrounding objects by monitoring surroundings using surrounding images and navigation sensors. The system may also include a remote navigation controller configured to support proximity situation recognition of the unmanned surface vehicle according to detection of the surrounding objects, wherein the unmanned surface vehicle may include an image acquisition processor, a navigation sensor, and a detector.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority to Korean Patent Application No.10-2021-0167292, filed Nov. 29, 2021, the entire contents of which isincorporated herein for all purposes by this reference.

BACKGROUND Technical Field

The present disclosure relates to a system and method formulti-image-based vessel proximity situation recognition support.

Description of Related Technology

An unmanned ship is a ship that automatically navigates a set routewithout a crew member, and refers to a vessel whose navigation andengine parts (e.g. engine, rudder) can be controlled from a remotecontrol center, if necessary. To this end, a remote control center isneeded on shore to control the unmanned ship remotely, and a qualifiedperson must directly conduct command and control at the remote controlcenter in order to resolve technical and legal issues.

SUMMARY

One aspect is a system and method for multi-image-based vessel proximitysituation recognition support that detects obstacles around an unmannedsurface vehicle (USV) using multiple cameras and various navigationsensors mounted on the USV, and provides remote situational awarenessinformation regarding the risk of collision with the detected obstacles.

Another aspect is a system for multi-image-based vessel proximitysituation recognition support.

In some embodiments, the system includes: an unmanned surface vehicle(USV) configured to detect and track surrounding objects by monitoringsurroundings using surrounding images and navigation sensors; and aremote navigation control device configured to support proximitysituation recognition of the unmanned surface vehicle according todetection of the surrounding objects, wherein the unmanned surfacevehicle may include: an image acquisition unit configured to acquiremultiple images showing the surroundings of the unmanned surface vehicleto detect objects around the unmanned surface vehicle through imageanalysis; a navigation sensor unit configured to acquire currentnavigation information of the unmanned surface vehicle and informationon obstacles around the unmanned surface vehicle in real time; and adetection unit configured to monitor the surroundings of the unmannedsurface vehicle by using the multiple images, current navigationinformation, and information on obstacles and, when an object close tothe unmanned surface vehicle within a preset distance is detected as aresult of monitoring, track the object detected.

The image acquisition unit may include: a thermal imaging camera, apanoramic camera, and a 360-degree camera installed to photograph thesurroundings of the unmanned surface vehicle, and acquires the multipleimages including thermal images, panoramic images; and 360-degree imagesof the surroundings of the unmanned surface vehicle in a form of AroundView (AV).

The navigation sensor unit may include: a global positioning system(GPS), a gyro sensor, an automatic identification system (AIS), andRADAR, wherein the GPS and gyro sensor may obtain the current navigationinformation including location, speed, direction, and posture of theunmanned surface vehicle, while the AIS and RADAR may obtain theinformation on obstacles around the unmanned surface vehicle.

The remote navigation control device may include: an estimation unitconfigured to estimate a collision risk between the unmanned surfacevehicle and the object detected by using the multiple images, currentnavigation information, and information on obstacles; and a situationrecognition unit configured to display the collision risk estimated,along with the object detected, on an electronic navigational chart foreach detected object and to output the collision risk in order tosupport proximity situation recognition.

The estimation unit may determine whether the object detected is locatedon an expected path of the unmanned surface vehicle by using the currentnavigation information, and when it is determined that the objectdetected is located on the expected path, may calculate the collisionrisk between the unmanned surface vehicle and the object detected, byusing fuzzy inference.

The situation recognition unit may display the collision risk for theobject detected in an augmented reality (AR) or virtual reality (VR)screen based on the multiple images and may output the screen in orderto support the proximity situation recognition.

Another aspect is a method for multi-image-based vessel proximitysituation recognition support performed in a system including anunmanned surface vehicle and a remote navigation control device.

In some embodiments, the method may include: acquiring, by the unmannedsurface vehicle, multiple images of surroundings to detect surroundingobjects through image analysis; obtaining, by the unmanned surfacevehicle, current navigation information and information on surroundingobstacles in real time; monitoring, by the unmanned surface vehicle, thesurroundings using the multiple images, current navigation information,and information on surrounding obstacles and, when an object close to asurrounding area within a preset distance is detected as a result ofmonitoring, tracking the object detected; and supporting, by the remotenavigation control device, proximity situation recognition of theunmanned surface vehicle according to detection of the object.

As described above, a system and method for multi-image-based vesselproximity situation recognition support according to an embodiment ofthe present disclosure has an effect that collision accidents with shipsand obstacles in proximity can be prevented in advance during remoteoperation of an unmanned surface vehicle (USV) by detecting obstaclesaround the unmanned surface vehicle using multiple cameras and variousnavigation sensors mounted on the unmanned surface vehicle, andproviding remote situational awareness information regarding the risk ofcollision with the detected obstacles.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objectives, features, and other advantages of thepresent disclosure will be more clearly understood from the followingdetailed description when taken in conjunction with the accompanyingdrawings.

FIG. 1 is a view schematically illustrating an environment in which asystem and method for multi-image-based vessel proximity situationrecognition support according to an embodiment of the present disclosuremay be implemented.

FIGS. 2 and 3 are views schematically illustrating the configuration ofthe system and method for multi-image-based vessel proximity situationrecognition support according to the embodiment of the presentdisclosure.

FIGS. 4 and 5 are views illustrating the system and method formulti-image-based vessel proximity situation recognition supportaccording to the embodiment of the present disclosure.

FIG. 6 is a flowchart schematically illustrating an operating method ofa system for multi-image-based vessel proximity situation recognitionsupport according to the embodiment of the present disclosure.

DETAILED DESCRIPTION

Due to the recent development of GPS and a variety of sensors, it is notdifficult for a vehicle to autonomously drive on the shortest route, butit is not so in the case of ship navigation. This is because, unlikeland-based vehicles, such as cars and motorcycles, ships have very largeinertial forces due to the nature of being operated on water, and thusit is very difficult for ships to instantly adjust their speed ordirection. In addition, the reality is that it is very difficult tonavigate on a scheduled route at sea because the sea does not have afixed road like on land and is greatly affected by multiple variablessuch as weather. Moreover, crew members should always look ahead toprevent unexpected collisions with other ships and reefs.

Particularly, in case of heavy rain or snow, high concentration of fog,smog, yellow dust or fine dust during ship operation, visibility israpidly reduced, making it difficult for the crew members to detectobjects with the naked eye even if they look ahead. Consequently, theship and its crew members may be put in very dangerous situations wherethey may collide with other ships and reefs unexpectedly, which isproblematic. In other words, when it comes to ship navigation, collisionavoidance is essential, and in order to adjust the speed or direction ofa ship, it is necessary to predict the ship's navigation path in advanceand operate a steering wheel or shift lever in advance.

As used herein, the singular expression includes the plural expressionunless the context clearly dictates otherwise. In this specification,terms such as “comprise”, “include”, or “have” should not be construedas necessarily including all of the various components or stepsdescribed in the specification, but should be construed that some of thecomponents or steps may not be included, or may further includeadditional components or steps. In addition, terms such as “. . . unit”,“. . . part”, and “module” described in the specification mean a unitthat processes at least one function or operation, which may beimplemented as hardware or software or a combination thereof.

Hereinafter, various embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings.

FIG. 1 is a view schematically illustrating an environment in which asystem and method for multi-image-based vessel proximity situationrecognition support according to an embodiment of the present disclosuremay be implemented, FIGS. 2 and 3 are views schematically illustratingthe configuration of the system and method for multi-image-based vesselproximity situation recognition support according to the embodiment ofthe present disclosure, and FIGS. 4 and 5 are views illustrating thesystem and method for multi-image-based vessel proximity situationrecognition support according to the embodiment of the presentdisclosure. Hereinafter, with reference to FIGS. 1 to 5 , the system formulti-image-based vessel proximity situation recognition supportaccording to the embodiment of the present disclosure will be described.

Referring to FIG. 1 , the system for multi-image-based vessel proximitysituation recognition support according to the embodiment of the presentdisclosure may be implemented in an unmanned surface vehicle (USV) andan onshore control center 200.

For example, communication between the unmanned surface vehicle 100 andthe onshore control center 200 may be performed through a marinecommunication network such as LTE-Maritime or a communication satellite.

Referring to FIG. 2 , the system for multi-image-based vessel proximitysituation recognition support according to the embodiment of the presentdisclosure may be configured to include a remote navigation controldevice (or a remote navigation controller) 210 installed in the unmannedsurface vehicle 100 and the onshore control center 200.

The unmanned surface vehicle 100 detects and tracks surrounding objectsby monitoring the surroundings using surrounding images and navigationsensors.

The remote navigation control device 210 supports the recognition ofsurrounding situations by the unmanned surface vehicle in line with thedetection of surrounding objects.

As shown in FIG. 2 , the unmanned surface vehicle 100 may include acommunication unit 110, a ship operation control unit 120, an imageacquisition unit (or an image acquisition processor) 130, a navigationsensor unit (or a navigation sensor) 140 and a detection unit (or adetector) 150.

The communication unit 110 communicates with other ships or land-basedcommunicable devices. For example, the communication unit 110 mayinclude various communication media such as CDMA, satellitecommunication, LTE, and RF communication.

In particular, the communication unit 110 may communicate with theremote navigation control device 210 installed in the onshore controlcenter 200 that remotely supports navigation of ships.

The ship operation control unit 120 controls the operation of theunmanned surface vehicle 100 according to control commands and controlinformation received from the remote navigation control device 210 inthe onshore control center 200.

That is, the remote navigation control device 210 may receive controlcommands and control information from a user and transmit the receivedcontrol commands and control information to the unmanned surface vehicle100.

Meanwhile, the ship operation control unit 120 may have an autonomousnavigation function.

For example, the ship operation control unit 120 may receive destinationinformation, generate an optimal navigation route to the destination,and control the unmanned surface vehicle 100 to navigate according tothe generated optimal navigation route. The ship operation control unit120 may be implemented as software, hardware, or a combination thereof,and may include an electronic navigational chart database, a routealgorithm for calculating an optimal navigation route, and the like.

In addition, the ship operation control unit 120 may control the speedand direction of the unmanned surface vehicle 100 by controlling theengine and steering gear of the unmanned surface vehicle 100. At thistime, the ship operation control unit 120 may control the speed anddirection of the unmanned surface vehicle 100 so that the unmannedsurface vehicle 100 avoids surrounding ships or obstacles by utilizingthe image acquisition unit 130 and the navigation sensor unit 140 to bedescribed later to monitor the surrounding situations.

The image acquisition unit 130 acquires multiple images showing thesurroundings of the unmanned surface vehicle 100 in order to detectobjects around the unmanned surface vehicle 100 through image analysis.

That is, the image acquisition unit 130 may include a thermal imagingcamera, a panoramic camera, and a 360-degree camera installed tophotograph the surroundings of the unmanned surface vehicle 100. Assuch, the image acquisition unit 130 may acquire thermal images,panoramic images, and 360-degree images showing the surroundings of theunmanned surface vehicle 100 in the form of Around View (AV).

The navigation sensor unit 140 acquires current navigation informationof the unmanned surface vehicle 100 and information about obstaclesaround the unmanned surface vehicle 100 in real time.

That is, the navigation sensor unit 140 may include a global positioningsystem (GPS), a gyro sensor, an automatic identification system (AIS),RADAR, LiDAR, and the like.

The GPS and gyro sensor may obtain current navigation informationincluding the location, speed, direction, and posture of the unmannedsurface vehicle 100.

The AIS, RADAR, and LiDAR may obtain information about obstacles aroundthe unmanned surface vehicle 100. The obstacle information may includeinformation on an unidentified object (obstacle) located around theunmanned surface vehicle 100 or a ship operating around the unmannedsurface vehicle 100.

That is, the AIS receives and collects AIS data, which is the track dataof ships around the unmanned surface vehicle 100. At this time, the AISdata consists of static date and dynamic data. The static date includesinformation on ship name, specifications, and destination while thedynamic data includes navigation information such as the currentlocation, course, and speed of a ship.

The RADAR and LiDAR detect objects existing around the unmanned surfacevehicle 100, and obtain location information, movement speed, movementdirection, and shape information of the detected objects. At this time,the objects may include a ship sailing around the unmanned surfacevehicle 100, an iceberg, a reef, a floating object, and the like.

By using the multiple images showing the surroundings of the unmannedsurface vehicle 100, as well as obstacle information and currentnavigation information acquired by the image acquisition unit 110 andthe navigation sensor unit 140, the detection unit 150 monitors thesurroundings of the unmanned surface vehicle 100 and, when an objectclose to the unmanned surface vehicle 100 within a preset distance isdetected as a result of monitoring, tracks the detected object.

At this time, the detection unit 150 may transmit the multiple-imageinformation, obstacle information, and current navigation informationacquired by the image acquisition unit 110 and the navigation sensorunit 140 to the remote navigation control device 210, and in case anobject is detected, may also transmit information on the detected objectto the remote navigation control device 210.

Referring to FIG. 3 , the remote navigation control device 210 may beconfigured to include a communication unit 211, an interface unit 212,an estimation unit (or an estimation processor) 213 and a situationrecognition unit (or a situation recognition processor) 214.

The communication unit 211 communicates with various ships in operation.For example, the communication unit 211 may include variouscommunication media such as CDMA, satellite communication, LTE, and RFcommunication.

In particular, the communication unit 211 may communicate with theunmanned surface vehicle 100 that a remote operator wants to controlthrough the remote navigation control device 210.

The interface unit 212 is a means for the remote operator in the onshorecontrol center 200 to control the unmanned surface vehicle 100 locatedremotely.

For example, the interface unit 212 may include a display, a remotecontroller, and the like. The display may output the multiple imagesshowing the surroundings of the unmanned surface vehicle 100, as well asobstacle information and current navigation information acquired by theimage acquisition unit 110 and the navigation sensor unit 140 andreceived from the unmanned surface vehicle 100. In addition, the remotecontroller is a device for the remote operator to control the unmannedsurface vehicle 100, and may receive various control commands for theunmanned surface vehicle 100 and transmit the received control commandsto the unmanned surface vehicle 100.

The estimation unit 213 estimates a collision risk between the unmannedsurface vehicle 100 and an object around the unmanned surface vehicle100 detected by the detection unit 150 by using the multiple-imageinformation, obstacle information, and current navigation informationreceived from the unmanned surface vehicle 100.

For example, the estimation unit 213 may calculate the collision risk byusing fuzzy inference.

That is, the estimation unit 213 may determine whether the detectedobject is located on the expected path of the unmanned surface vehicle100 by using the current navigation information including the location,speed, direction, and posture of the unmanned surface vehicle 100 and,when it is determined that the detected object is located on theexpected path, may calculate the collision risk between the unmannedsurface vehicle 100 and the detected object, by using fuzzy inference.

In addition, as a result of tracking by the detection unit 150, when thedetected object is identified as a moving object such as a ship, theestimation unit 213 may calculate the time when the detected objectpasses through the expected path of the unmanned surface vehicle 100,and calculate the collision risk between the unmanned surface vehicle100 and the detected object at the time of passing through thecalculated predicted path by the detected object, by using fuzzyinference.

The situation recognition unit 214 displays the collision riskcalculated by the estimation unit 213 through the display of theinterface unit 212 along with the detected object on an electronicnavigational chart for each detected object and outputs the displayedcollision risk in order to support the remote operator's awareness ofthe surrounding situations of the unmanned surface vehicle 100.

In addition, the situation recognition unit 214 may display thecalculated collision risk for the detected object in an augmentedreality (AR) or virtual reality (VR) screen based on the multiple imagesshowing the surroundings of the unmanned surface vehicle 100 and outputthe screen in order to support the remote operator's awareness of thesurrounding situations of the unmanned surface vehicle 100.

For example, as shown in FIGS. 4 and 5 , on the augmented reality orvirtual reality-based surrounding situation recognition support screenthat is output, detected objects are highlighted to increase theirvisibility, and among the detected objects, the one with a high risk ofcollision due to close proximity to the unmanned surface vehicle 100 maybe displayed with a sign indicating the danger. FIG. 4 shows anaugmented reality and virtual reality-based surrounding situationrecognition support screen using 360-degree images, and FIG. 5 shows anaugmented reality-based proximity situation recognition support screenusing a panoramic image.

FIG. 6 is a flowchart schematically illustrating an operating method ofa system for multi-image-based vessel proximity situation recognitionsupport according to the embodiment of the present disclosure.

In the step of S610, the unmanned surface vehicle 100 acquires multipleimages of the surroundings by utilizing the image acquisition unit 130and the navigation sensor unit 140, obtains current navigationinformation of the unmanned surface vehicle 100 and information onobstacles around the unmanned surface vehicle 100 in real time, andtransmits the acquired multi-image information, current navigationinformation, and obstacle information to the remote navigation controldevice 210.

In the step of S620, the unmanned surface vehicle 100 monitors thesurroundings using multiple images, current navigation information, andobstacle information, and as a result of monitoring, when an objectclose to the surrounding area within a preset distance is detected, theunmanned surface vehicle 100 tracks the detected object.

At this time, in case the object is detected, the unmanned surfacevehicle 100 may also transmit information on the detected object to theremote navigation control device 210.

In the step of S630, the remote navigation control device 210 estimatesa collision risk between the unmanned surface vehicle 100 and the objectaround the unmanned surface vehicle 100 detected by the detection unit150 by using the multiple-image information, obstacle information, andcurrent navigation information received from the unmanned surfacevehicle 100.

In the step of S640, the remote navigation control device 210 displaysthe estimated collision risk along with the object on the electronicnavigational chart for each detected object and outputs the displayedcollision risk in order to support the remote operator's awareness ofthe surrounding situations of the unmanned surface vehicle 100.

In addition, the remote navigation control device 210 may display andoutput the calculated collision risk for the detected object displayedon the augmented reality or virtual reality screen based on the multipleimages showing the surroundings of the unmanned surface vehicle 100 inorder to support the remote operator's awareness of the surroundingsituations of the unmanned surface vehicle 100.

Meanwhile, components of the above-described embodiment may be easilygrasped from a process perspective. That is, each component may beidentified as a separate process. In addition, the process of theabove-described embodiment may be easily grasped from the point of viewof components of the device.

Furthermore, the technical contents described above may be implementedin the form of program instructions that can be executed through variouscomputer means and recorded in a computer-readable medium. Thecomputer-readable medium may include program instructions, data files,data structures, etc., or a combination thereof. The programinstructions recorded on the medium may be specially designed andconfigured for the embodiments or may be known and usable to thoseskilled in the art of computer software. Examples of computer-readablemedium include magnetic storage media such as hard disks, floppy disks,and magnetic tapes, optical media such as CD-ROM and DVD,magneto-optical media, such as floptical disks, a hardware devicespecially configured to store and execute program instructions, such asROM, RAM, flash memory, etc. Examples of the program instructionsinclude high-level language codes that can be executed by a computerusing an interpreter, as well as machine language codes such as thoseproduced by a compiler. The hardware device may be configured to act asone or more software modules to perform the operations of theembodiments and vice versa.

The foregoing embodiments of the present disclosure have been disclosedfor purposes of illustration. Those skilled in the art with ordinaryknowledge of the present disclosure will be able to make variousmodifications, changes, and additions within the spirit and scope of thepresent disclosure, and these modifications, changes, and additionsshould be regarded as belonging to the scope of the following claims.

What is claimed is:
 1. A system for multi-image-based vessel proximitysituation recognition support, the system comprising: an unmannedsurface vehicle (USV) configured to detect and track surrounding objectsby monitoring surroundings using surrounding images and navigationsensors; and a remote navigation controller configured to supportproximity situation recognition of the unmanned surface vehicleaccording to detection of the surrounding objects, wherein the unmannedsurface vehicle comprises: an image acquisition processor configured toacquire multiple images showing the surroundings of the unmanned surfacevehicle to detect objects around the unmanned surface vehicle throughimage analysis; a navigation sensor configured to acquire currentnavigation information of the unmanned surface vehicle and informationon obstacles around the unmanned surface vehicle in real time; and adetector configured to monitor the surroundings of the unmanned surfacevehicle by using the multiple images, current navigation information,and information on obstacles and, when an object close to the unmannedsurface vehicle within a preset distance is detected as a result ofmonitoring, track the object detected.
 2. The system formulti-image-based vessel proximity situation recognition support ofclaim 1, wherein the image acquisition processor includes a thermalimaging camera, a panoramic camera, and a 360-degree camera installed tophotograph the surroundings of the unmanned surface vehicle, and whereinthe image acquisition processor is configured to acquire the multipleimages including thermal images, panoramic images, and 360-degree imagesof the surroundings of the unmanned surface vehicle in a form of AroundView (AV).
 3. The system for multi-image-based vessel proximitysituation recognition support of claim 1, wherein the navigation sensorincludes a global positioning system (GPS), a gyro sensor, an automaticidentification system (AIS), and RADAR, wherein the GPS and gyro sensorare configured to obtain the current navigation information includinglocation, speed, direction, and posture of the unmanned surface vehicle,while the AIS and RADAR obtain the information on obstacles around theunmanned surface vehicle.
 4. The system for multi-image-based vesselproximity situation recognition support of claim 1, wherein the remotenavigation controller comprises: an estimation processor configured toestimate a collision risk between the unmanned surface vehicle and theobject detected by using the multiple images, current navigationinformation, and information on obstacles; and a situation recognitionprocessor configured to display the collision risk estimated, along withthe object detected, on an electronic navigational chart for eachdetected object and to output the collision risk in order to supportproximity situation recognition.
 5. The system for multi-image-basedvessel proximity situation recognition support of claim 4, wherein theestimation processor is configured to determine whether the objectdetected is located on an expected path of the unmanned surface vehicleby using the current navigation information, and when it is determinedthat the object detected is located on the expected path, calculate thecollision risk between the unmanned surface vehicle and the objectdetected, by using fuzzy inference.
 6. The system for multi-image-basedvessel proximity situation recognition support of claim 4, wherein thesituation recognition processor is configured to display the collisionrisk for the object detected in an augmented reality (AR) or virtualreality (VR) screen based on the multiple images and outputs the screenin order to support the proximity situation recognition.
 7. A method formulti-image-based vessel proximity situation recognition supportperformed in a system including an unmanned surface vehicle and a remotenavigation control device, the method comprising: acquiring, by theunmanned surface vehicle, multiple images of surroundings to detectsurrounding objects through image analysis; obtaining, by the unmannedsurface vehicle, current navigation information and information onsurrounding obstacles in real time; monitoring, by the unmanned surfacevehicle, the surroundings using the multiple images, current navigationinformation, and information on surrounding obstacles and, when anobject close to a surrounding area within a preset distance is detectedas a result of monitoring, tracking the object detected; and supporting,by the remote navigation control device, proximity situation recognitionof the unmanned surface vehicle according to detection of the object.